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ABSTRACT 

The capability to remotely, robotically perform space assembly, inspection, servicing, and science functions would 
rapidly expand our presence in space, and the cost efficiency of being there. There is thus considerable interest in 
developing “telerobotic” technologies, which also have comparably important terrestrial applications to health care, 
underwater salvage, nuclear waste remediation and other. Such tasks, both space and terrestrial, require both a robot 
and operator interface that is highly flexible and adaptive, i.e., capable of efficiently working in changing and often 
casually structured environments. One systems approach to this requirement is to augment traditional teleoperation 
with computer assists - advanced teleoperation. We have spent a number of years pursuing this approach, and high- 
light some key technology developments and their potential commercial impact. This paper is an illustrative sum- 
mary rather than self-contained presentation; for completeness, we include representative technical references to our 
work which will allow the reader to follow up items of particular interest. 

A BRIEF TECHNICAL OVERVIEW 

Telerobotics technology development [ 1 ] is motivated by a desire to remotely perform complex physical tasks under 
human supervisory control. To date, robotic systems that have embodied significant supervisory (autonomous) con- 
trol of their manipulation functions have been limited to highly structured tasks that were performed under favorable 
and certain conditions - by definition not complex tasks, and not adaptive performance. This has fostered the wide- 
spread use of teleoperation, which at the other extreme from automation, is a characteristically laborious manual con- 
trol procedure, historically applied to hazardous environments such as nuclear materials handling, underseas recov- 
ery, and recently, space shuttle operations. Virtual environments and virtual reality (VR) engineering are related and 
currently popular areas of technology development, wherein the human operator directly manipulates or experiences 
a modeled, rather than physical reality via computer-synthesis and appropiate input/output devices (e.g., master con- 
trol gloves/stereo-immersive displays). There exists an important technical intersection of VR technology with teler- 
obotics, most specifically with teleoperation: Virtual environments are useful tools for simulation and design, inclu- 
ding task analysis, training, and on-line task preview and prediction. Thus, if VR can be efficiently integrated and 
physically calibrated with teleoperation systems, it has promise to assist the operator’s on-line perception, planning, 
and control functions. 

With regard to space applications, teleoperation systems could have important near-term roles in remote platform ser- 
vicing, telescience, and lunar exploration, as already illustrated in Shuttle STS-RMS operations. However, the physi- 
cal and logistical demands of space telemanipulation, particularly in less structured environments, will be high. Tasks 
can be physically complex and time-consuming, and the operator’s manual dexterity and hand-to-eye motion calibra- 
tion must be good. Further, the work will often be conducted under degraded observational conditions and thus be te- 
dious and fatiguing. Operational uncertainties include obstructed viewing and manipulation, as well as the very dis- 
orienting effects of possible time-delay between the operator inputs and robot actions (a major obstacle to achieving 
desirable ground versus on-orbit operations). In the face of these cdllective challenges (which have their metaphors in 
other applications areas such as minimally invasive medical robotics and deep sea teleoperations), we have been try- 
ing to “computer-enhance” the performance of traditional teleoperation, and have made progress in the technical ar- 
eas of redundant telemanipulator control, viewing systems, real-time graphics-based task simulation and predictive 
control integrated operator interface design, systems-scale ground laboratory experiments and accompanying human 
factors data collection & analysis. The laboratory photographs of the next page give a sense of our system implemen- 
tation; we comment below on our specific enabling technical advances (with supporting citations). For the reader 
seeking a detailed engineering overview of this work through end- 1991, see reference [2] 
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A main experimental thrust in our lab has been end-to-end system-level performance characterization - formal ex- 
periment design, integrated system demonstrations, task instrumention & data capture, and human factors analysis. 
Collectively, the goal has been to quantify operator limitations, component technology requirements, and their inter- 
dependencies in the context of tasks simulated with realistically posed operational constraints (variable lighting, task 
geometry, time-delay, control & communication bandwidths, viewing & display limitations, etc.). Accompanying 
human factors issues are the assessment of technology impact on operator error, workload, and training, each in itself 
a significant risk and cost driver for space operations. As noted above, advanced teleoperation is computer-assisted 
telemanipulation, wherein the operator retains manual control of the task, but with extended functional capabilities 
and reduced cognitive complexity of task interaction. The computer assists we have developed to date encompass in- 
teractive task planning/simulation aids [3], graphics user interfaces for system programming/command/status display 
[5], and several modes of force-referenced teloperator control which are tolerant to operator positioning error (e.g., 
“shared compliance control” as described in [2,7] and references therein). In its most general form, advanced teleop- 
eration entails sensory fusion and decentralized control, given that the system sensing, planning, and control func- 
tions are inherently distributed between operator and computer; to this end, we have developed some generalized 
control models and architectures for man-machine interaction at multiple levels of control abstraction, also related 
multisensor fusion models and techniques [6]. Regarding conventional controls, we have investigated a variety of ki- 
nesthetic position, rate, force-feedback, and shared compliance teleoperations modes [2,7]; these controls were first 
applied to dual six degree of freedom (d.o.f.) PUMA manipulators and more recently to high-dexterity eight d.o.f. re- 
dundant manipulators [8], whose controls development has included computer-based techniques of task redundancy 
management and visualization. We have quantitatively evaluated the operator utility of these these control modes, 
along with more traditional position and rate approaches, through simulated space servicing experiments [7], As one 
example, we performed human factors-based experiments which telerobotically re-enacted high dexterity Solar Max- 
imum Mission satellite repair procedures originally performed by astronaut extra- vehicular activity (EVA) during the 
1984 space shuttle flight STS-13. Other supporting developments include real-time graphics environments which al- 
low the operator to animate, analyze, and train on teleoperator tasks, and in a most general case, actually use the 
graphic virtual environment as a basis for reliable teleoperation under multiple second time delay [3,4], We believe 
the area of graphics-augmented reality for teleoperation has particular promise in space applications and comment 
further by way of an illustrative example. 


AN APPLICATION HIGHLIGHT 

A significant obstacle to the acceptance of space telerobotic systems is the impact they might have on operational 
timelines of crew and platform resources. If a significant part of this burden could be shifted to ground operations, 
then the technology benefits of space robotics would be far greater. Serendipitously, utilizing ground operations 
would also free the operator control station of many space-borne implementation constraints, e.g., high degrees of 
computational power could be brought to bear. However, ground operation of a space robot performing a complex 
task confronts a basic system limitation in that robotic automation is not yet sufficiently generalized to allow conven- 
tional missions control by uplink sequencing of discrete high-level commands. Rather, the operator’s continuous 
direct manual control and eye-to-hand perceptual coordination is required and unfortunately, the implied ground-to- 
orbit teleoperation approach will not alone suffice either. The problem lies in time-delay communications transit (2- 
10 seconds latency in current operations scenarios). The operator cannot “fly-by-wire” confidently or coordinate his 
eye-to-hand skills when causal action-reaction is on the order of seconds; indeed, people rapidly adopt a laborious 
move-and-wait behavioral pattern when round-loop control latencies are greater than .25 seconds. 


Our approach to resolving this fundamental limitation has been to develop a class of 3-D graphics display which visu- 
ally simulates the robot response in real-time immediacy to the operator’s input. In essence, the operator interacts 
with a virtual task model. Thus, the critical details of the task (and robot itself) must be accurately modeled, and fur- 
ther, must be very accurately geometrically calibrated to the operator’s time-delayed visual reality as displayed by 
down-linked video. In terms of practical implementation, this results in a 3-D high-fidelity graphics display which 
must be correctly registered and overlaid in translation, scale, and aspect re. a multi-camera robot workspace presen- 
tation. See the second page of laboratory photographs for a representative example. Our development of this predic- 
tive graphics display (with a calibrated virtual reality) has enabled us to preserve the operational features of 
teleoperation, and reliably operate with intermittent time delays up to 5-10 seconds. In a recent demonstration 
depicted in the lab photos, we, in coordination with colleagues at NASA Goddard Space Flight Center, performed a 
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(robot operator's multi-media display during task) 




simulated ground-to-remote on-orbit equipment changeout similar to that anticipated for future Hubble Space Tele- 
scope servicing: from JPL, having geometrically modeled and visually calibrated the “remote” GSFC robot site, we 
teleoperatively detached and remounted an ORU. The motion planning and execution, both in free space and 
guarded-contact, were generated by pure teleoperation, with accuracies of millimeters over a work volume of several 
meters cubed. 


COMMERCIAL MARKETS 

The ability to calibrate and animate a virtual environment with respect to actual visual robotic workspaces appears to 
have significant applications potential. As one example, in the area of medical robotics, it suggests a number of possi- 
bilities for computer-guided stereotaxic procedures, microtelerobotic surgery, telesurgery proper (actual remote surgi- 
cal theatres), also multisensory data presentation and visualization. And of course, calibrated VR seemingly is a key 
ingredient in planning and executing telerobotic operations in remote scenarios subject to either time delay and or 
partial viewing obstruction. To this end we have joined with Deneb Robotics, Inc., of Auburn Hills, MI, to coopera- 
tively develop a calibrated 3-D graphics-on-video function within their line of 3-D graphics simulation products. 
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